Semantic column-type inference + override UI [prototype]#144
Draft
jeanbez wants to merge 3 commits into
Draft
Conversation
Prototype: classify each column as continuous / categorical / identifier by dtype + cardinality (+ id-name hint) instead of raw dtype, so numeric codes (e.g. class labels) and identifiers (e.g. unique keys) aren't summarized as continuous measurements. - infer_column_roles(df, overrides) helper in web/routes/utils.py - /summary-statistics: describe() over continuous only; value-count bars for categoricals; identifiers listed and excluded from stats/plots - /column-roles endpoint persists per-session overrides; roles reset on new upload; overrides bust cached metric results - Data Overview 'Override Column Types' panel: per-column role dropdowns with search + 14-row cap for wide datasets, Apply recomputes - unit tests for the inference heuristic (incl. small-dataset guards) Web-only for now; CLI/Globus/library unaffected (see follow-ups).
jeanbez
changed the title
…sting E231 - utils.py: move multi-line threshold comments to own lines (E114/E116) - inspector.js: prettier --write formatting - retriever.py: add missing space after comma (E231, pre-existing on develop)
Collaborator
Author
|
@kaveenh this is the draft of that feature we discussed, please go ahead and see what needs changed and replicated to the other interfaces |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Metrics classify columns by raw pandas dtype, so numeric columns that are really categorical codes (e.g. a class label
flavor= -1/1/2/5/21) or identifiers (e.g. a near-uniquejetIndex) get summarized as continuous measurements — meaningless means/percentiles, KDE plots, and correlations.What this does
Introduces a semantic role per column —
continuous/categorical/identifier— inferred from dtype + cardinality (+ a light id-name hint), with a user override.infer_column_roles(df, overrides)helper (web/routes/utils.py) — cardinality/name heuristic with small-dataset guards; overrides win./summary-statistics—describe()over continuous columns only; value-count bar charts for categoricals (a KDE of a discrete code is meaningless); identifiers listed and excluded from stats/plots./column-rolesendpoint — persists per-session overrides; roles reset on new upload; applying an override busts cached metric results.Scope / follow-ups (not in this PR)
web/), so no regression — but they don't benefit yet and now differ from the web UI.aidrin/package so all surfaces share one source of truth; add string-identifier detection (UUIDs/hashes); wire the metrics themselves to consume roles (esp. privacy, where identifiers are input, not noise); and address distributions-at-scale (100 cols → 90 plots on one page).Testing